Conceptualization is a critical process for individuals in a variety of professional, technical, and academic fields. Designers, architects, engineers, medical professionals, animators and teachers are only some groups of individuals who benefit greatly from systems and methods designed to facilitate the conceptualization process.
Traditionally, conceptualization makes heavy use of drawing and sketching as aids to visual thinking, due to their simplicity and fluidity. Professionals will often create multiple sketches, highlighting certain properties, views, and/or ideas of a structure or concept. Typically, these sketches are 2D representations of a 3D idea; however there is no defined “middle ground” or information that “goes between” the sketch and the object. In fact, a crucial stage in the conceptualization process is the effective mental fusion of a collection of these sketches, to arrive at a more coherent and complete visualization of an idea. However, the roughness and incompleteness of conceptual sketches often make it difficult to resolve ambiguities without further input from the user, and it is difficult to provide an intuitive way to convey that information. These challenges have been passed over by 3D modeling and visualization programs that instead support the modeling of well-defined geometric objects. Beyond conceptualizing for the purpose of one's own personal knowledge, sketches are often used as aids to instruction, communication, and organization.
A computer allows 3D rather than 2D sketching. The ability to change viewpoint dynamically has been shown to enhance the perception of 3D structures or concepts compared to the mental merging of static views. (T. Sando, M. Tory, and P. Irani, “Effects Of Animation, User-Controlled Interactions, And Multiple Static Views In Understanding 3D Structures,” In Proc. Applied Perception in Graphics and Visualization, ACM, pages 69-76, 2009; R. L. Sollenberger, and P. Milgram, “Effects Of Stereoscopic And Rotational Displays In A Three-Dimensional Pathtracing Task,” Human Factors 35, 3, pages 483-499, 1993).
The earliest computer-based sketching system, known as “Sketchpad,” was developed in the early 1980s by Ivan Sutherland (I. E. Sutherland, “Sketchpad: A Man-Machine Graphical Communication System,” New York: Garland Publishers, 1980), and followed by Sachs et al.'s creation of 3-draw (E. Sachs, A. Roberts, and D. Stoops, “3-Draw: A Tool For Designing 3D Shapes,” IEEE Comput. Graph. Appl., 11, 6, pages 18-26, 1991), which introduced 3D sketching to the computer graphics community. Of note are systems such as Robert Zeleznik's SKETCH (R. C. Zeleznik, K. P. Herndon, and J. F. Hughes, “Sketch: An Interface For Sketching 3D Scenes,” In SIGGRAPH '96, pages 163-170, 1996), and Takeo Igarashi's Teddy (T. Igarashi, S. Matsuoka, and H. Tanaka, “Teddy: A Sketching Interface For 3D Freeform Design,” In SIGGRAPH '99, pages 409-416, 1999), both early attempts at inferring 3D geometry from pen-based user gestures. One problem with prior art techniques is that they are restrictive in that they require an explicitly-defined geometry at any given time, which often impedes rapid ideation and limits freedom in expressing and exploring forms. Systems such as Harold (J. M. Cohen, J. F. Hughes, and R. C. Zeleznik, “Harold: A World Made Of Drawings,” In Proc. Of The Symposium On Nonphotorealistic Animation And Rendering (NPAR), pages 83-90, 2000), and ILoveSketch (S-H. Bae, R. Balakrishnan, and K. Singh, “ILoveSketch: As-Natural-As-Possible Sketching System For Creating 3D Curve Models,” In Proceedings Of The 21st Annual ACM Symposium On User Interface Software And Technology (UIST '08), pages 151-160, 2008) explore interesting ideas of 3D curve sketching, either by imposing certain constraints on the strokes, or asking users to draw each stroke in two steps. While this can create some interesting sketches, the sketching process is again less fluid and more constrained than in traditional sketching. Other projective-based stroke systems (K. Kallio, “3D6B Editor: Projective 3D Sketching With Line-Based Rendering,” Proc. of Eurographics Workshop on Sketch-based Interfaces and Modeling, pages 73-79, 2005; O. Tolba, J. Dorsey, and L. McMillan, “A Projective Drawing System,” In Proc. of Symposium on Interactive 3D graphics (SI3D), pages 25-34, 2001) offer interesting notions of how to effectively place 2D sketched input fluidly into a 3D scene, but with significant geometric and interactive limitations. In summary, 3D sketching systems have tried to leverage the flexibility and intuitiveness of 2D sketching, while simultaneously providing a way to add depth and dimensionality to sketches, so they may be visualized to some degree in three dimensions. However, their inconsistent and often unwanted shape inference and interpretation of depth from hand-drawn input, their imposed constraints of the input, and their lack of straightforward, fluid interfaces limits their scope and use. As such, no widespread, commercially viable 3D sketching system currently exists.
In addition to sketches and drawings, many of the aforementioned user groups traditionally use images and photographs as supplementary material in creating, understanding and conveying a concept. For example, photos and images of existing relevant structures may serve as inspiration for a user. Engineers and architects often need photographs of structures to be able to design and ideate in relation to surrounding structures. Photos and images can be used by teachers as instructional tools to convey certain ideas. Medical professionals use images to help patients and other doctors understand the characteristics of anatomical and chemical processes and conditions. Despite their widespread use in concept visualizations, a similar problem exists as in 2D sketching, in that photos and images tend to be 2D visualizations of 3D objects or scenes, and the lack of dimensionality limits the understanding, organization and mental fusion of these visual aides.
Image collections allow for virtual tours of sites. One such system is Microsoft Photosynth, Microsoft, Inc., Redmond, Wash., USA, which allows for the organization of sets of photographs in 3D of an existing location, via estimating camera positions with bundling techniques, as described by Snavely et al. (N. Snavely, S. M. Seitz, and R. Szeliski, “Phototourism: Exploring Photo Collections In 3D,” ACM Trans. Graph 25, 3, pages 835-846, 2006). On the other end of the spectrum, entire detailed structures can also be reconstructed to some extent from images, using the work of Pollefeys et al. (M. Pollefeys, L. J. V. Gool, M. Vergauwen, F. Verbeist, K. Cornelis, J. Tops, and R. Koch, “Visual Modeling With A Hand-Held Camera,” Int. J. Computer Vision 59, 3, pages 207-232, 2004). To provide representations closer to 3D models, techniques have been explored such as automatically creating photo pop-ups (coarse, texture mapped geometry) from single photographs. The work of Ventura et al. (J. Ventura, S. Diverdi, and T. Hollerer, “A Sketch-Based Interface For Photo Pop-Up,” In Proc. Eurographics Symposium on Sketch-Based Interfaces and Modeling, pages 21-28, 2009) builds on this, by adding some user-flexibility, and designing an interface for specifying occlusion boundaries. These tools allow for the creation of 2D image-based or 3D model-based representations of scenes and can assist in viewing existing content, but they do not support the conceptualization or ideation process.
More generally, computers have recently proven critical in the visualization of real-world and/or virtual objects. Such visualizations can contribute greatly to a multitude of fields, including but not limited to, art, printed media, design, technology, medicine, automotive design, video games, films, television programs, commercials, etc. Visualizations that can be created by computers can be dynamic or static, and can be 2D or 3D. The images can be animated and can be rendered as a movie. Such images, whether static or dynamic, can be created virtually, where a virtual world can refer to an interactive environment.
Computer graphics software can be used to create the above visualizations. The availability of such computer graphics software and increased computer speeds have allowed users to produce high quality, professional-grade images, films, games, fine art, etc. using their computers.
Physical or virtual (that might not exist in a real world) objects can be represented or otherwise sketched using two broad classes of authoring tools that are available in computer graphics software. Such tools include 2D drawing and image-editing tools, such as Adobe Photoshop®, by Adobe Systems Incorporated, Mountain View, Calif., USA, and Autodesk Sketchbook Pro®, by Autodesk, Inc., San Rafael, Calif., USA, and 3D modeling packages, such as AutoCAD®, Maya®, and Revit®, all by Autodesk, Inc. Both categories of tools, while providing users with the ability to create respectively 2D images or 3D models, have significant limitations. In particular, using 2D sketching, an object can be created very fluidly and expressively, but the sketch is 2D and static. Three-dimensional modeling packages, while allowing the user to see the object from multiple vantage or viewing points, can be cumbersome, rigid and can further inhibit creativity, as the user may be required to strictly conform to specifics of the objects and/or limitations of the package and/or system that is running the package.
Thus, there is a need for a system and method that encompasses the ease of use and flexibility of sketching and imaging tools, while at the same time allowing for a clearer understanding of 3D structure and form. In other words, there is a need for a tool for generating and manipulating collections of 2D content that can then be progressively arranged and fused together in 3D space, to aid in understanding and visualizing concepts in three dimensions.
Attempts have been made in the research community to integrate a sketching system with imaging capabilities. Kalnins et al. (R. D. Kalnins, L. Markosian, B. J. Meier, M. A. Kowalski, J. C. Lee, P. L. Davidson, M. Webb, J. F. Hughes, and A. Finkelstein, “WYSIWYG NPR: Drawing Strokes Directly On 3D Models,” ACM Trans. on Graph. 21, 3, pages 755-762, 2002) developed a system to annotate existing 3D models with hand-drawn, non-photorealistically rendered (“NPR”) strokes in 3D. This offers the ability to add a personal rendering aesthetic to a 3D object. Tsang et al. (S. Tsang, R. Balakrishnan, K. Singh, and A. Ranjan, “A Suggestive Interface For Image Guided 3D Sketching,” In Proc. Of The SIGCHI Conference On Human Factors In Computing Systems (CHI), pages 591-598, 2004) introduced an image-aided sketching interface system, where 2D images are overlaid to guide user strokes. The system then produces a coarse, structured, 3D wireframe model. Lau et al. (M. Lau, G. Saul, J. Mitani, and T. Igarashi, “Modeling-In-Context: User Design Of Complementary Objects With A Single Photo,” In Proc. Symposium On Sketch-Based Interfaces And Modeling, pages 1-8, 2010) developed a system for sketches and markups on a single photo to define a 3D object. Insitu (P. Paczkowski, M. H. Kim, Y. Morvan, J. Dorsey, H. Rushmeier, and C. O'Sullivan, “Insitu: Sketching Architectural Designs In Context,” In Proceedings Of The 2011 SIGGRAPH Asia Conference (SA '11), 2011), an architectural design system, integrates a sketching system with a novel lightweight, environment site representation methodology, to conceptualize architectural scenes in context. These and other existing tools, however, are too specialized, too complex for broader use, or not suitable for true conceptualization of ideas that are not yet fully developed.
The consistent property of two-dimensionality of user-generated content used in the conceptualization process lends itself to the idea of a plurality of 2D planes that can be positioned in a 3D space. The 2D planes can include a variety of content, including, but not limited to, sketches, photographs, text, videos, and any other content having various levels of detail. Initially, the content can be generated without any specification of 3D positions. The content included in the 2D planes and/or the 2D planes themselves can be manipulated in the 3D space. The 2D planes along with their respective content then can be arranged in a 3D space to generate a 3D representation of a scene. The scene can correspond to a real-world physical object and/or collection of objects, or it can be a virtual scene, or it can be a combination of both. As the scene does not require an explicit geometric representation, the 2D content admits ambiguities, inconsistencies and incompleteness. Further, the system and methods can also provide various graphical tools that can allow users to perform various manipulations of the 2D planes, their content, and/or both.